Efficient approximation of optimal control for continuous-time Markov games

We study the time-bounded reachability problem for continuous-time Markov decision processes (CTMDPs) and games (CTMGs). Existing techniques for this problem use discretisation techniques to partition time into discrete intervals of size ε, and optimal control is approximated for each interval separately. Current techniques provide an accuracy of on each interval, which leads to an infeasibly large number of intervals. We propose a sequence of approximations that achieve accuracies of , , and , that allow us to drastically reduce the number of intervals that are considered. For CTMDPs, the performance of the resulting algorithms is comparable to the heuristic approach given by Buchholz and Schulz, while also being theoretically justified. All of our results generalise to CTMGs, where our results yield the first practically implementable algorithms for this problem. We also provide memoryless strategies for both players that achieve similar error bounds

PrIC3: Property Directed Reachability for MDPs

IC3 has been a leap forward in symbolic model checking. This paper proposes PrIC3 (pronounced pricy-three), a conservative extension of IC3 to symbolic model checking of MDPs. Our main focus is to develop the theory underlying PrIC3. Alongside, we present a first implementation of PrIC3 including the key ingredients from IC3 such as generalization, repushing, and propagation

Policy learning for time-bounded reachability in Continuous-Time Markov Decision Processes via doubly-stochastic gradient ascent

Continuous-time Markov decision processes are an important class of models in a wide range of applications, ranging from cyber-physical systems to synthetic biology. A central problem is how to devise a policy to control the system in order to maximise the probability of satisfying a set of temporal logic specifications. Here we present a novel approach based on statistical model checking and an unbiased estimation of a functional gradient in the space of possible policies. The statistical approach has several advantages over conventional approaches based on uniformisation, as it can also be applied when the model is replaced by a black box, and does not suffer from state-space explosion. The use of a stochastic gradient to guide our search considerably improves the efficiency of learning policies. We demonstrate the method on a proof-of-principle non-linear population model, showing strong performance in a non-trivial task

Understanding and Extending Incremental Determinization for 2QBF

Incremental determinization is a recently proposed algorithm for solving quantified Boolean formulas with one quantifier alternation. In this paper, we formalize incremental determinization as a set of inference rules to help understand the design space of similar algorithms. We then present additional inference rules that extend incremental determinization in two ways. The first extension integrates the popular CEGAR principle and the second extension allows us to analyze different cases in isolation. The experimental evaluation demonstrates that the extensions significantly improve the performance

Predictors of pain intensity and persistence in a prospective Italian cohort of patients with herpes zoster: relevance of smoking, trauma and antiviral therapy

Herpes zoster (HZ) is a common disease, characterized by rash-associated localized pain. Its main complication, post-herpetic neuralgia (PHN), is difficult to treat and may last for months to years in the wake of rash resolution. Uncertainties remain as to the knowledge of predictors of HZ-related pain, including the role of antiviral therapy in preventing PHN in ordinary clinical practice. This prospective cohort study was aimed at investigating pain intensity at HZ presentation and its correlates, as well as the incidence of PHN and its predictors

The cognitive neuroscience of prehension: recent developments

Prehension, the capacity to reach and grasp, is the key behavior that allows humans to change their environment. It continues to serve as a remarkable experimental test case for probing the cognitive architecture of goal-oriented action. This review focuses on recent experimental evidence that enhances or modifies how we might conceptualize the neural substrates of prehension. Emphasis is placed on studies that consider how precision grasps are selected and transformed into motor commands. Then, the mechanisms that extract action relevant information from vision and touch are considered. These include consideration of how parallel perceptual networks within parietal cortex, along with the ventral stream, are connected and share information to achieve common motor goals. On-line control of grasping action is discussed within a state estimation framework. The review ends with a consideration about how prehension fits within larger action repertoires that solve more complex goals and the possible cortical architectures needed to organize these actions

Managing multimorbidity in primary care in patients with chronic respiratory conditions

The term multimorbidity is usually defined as the coexistence of two or more chronic conditions within an individual, whereas the term comorbidity traditionally describes patients with an index condition and one or more additional conditions. Multimorbidity of chronic conditions markedly worsens outcomes in patients, increases treatment burden and increases health service costs. Although patients with chronic respiratory disease often have physical comorbidities, they also commonly experience psychological problems such as depression and anxiety. Multimorbidity is associated with increased health-care utilisation and specifically with an increased number of prescription drugs in individuals with multiple chronic conditions such as chronic obstructive pulmonary disease. This npj Primary Care Respiratory Medicine Education Section case study involves a patient in a primary care consultation presenting several common diseases prevalent in people of this age. The patient takes nine different drugs at this moment, one or more pills for each condition, which amounts to polypharmacy. The problems related with polypharmacy recommend that a routine medication review by primary care physicians be performed to reduce the risk of adverse effects of polypharmacy among those with multiple chronic conditions. The primary care physician has the challenging role of integrating all of the clinical problems affecting the patient and reviewing all medicaments (including over-the-counter medications) taken by the patient at any point in time, and has the has the key to prevent the unwanted consequences of polypharmacy. Multimorbid chronic disease management can be achieved with the use of care planning, unified disease templates, use of information technology with appointment reminders and with the help of the wider primary care and community teams